WEM Process Monitor

 

The Process Monitor is a script used for monitoring and maintaining WEM application software processes. It provides the flexibility of being run as a stand-alone program or a fully functional background daemon and is capable of logging to syslog and log file with customizable e-mail notification facilities. The ability to monitor and maintain these processes coupled with the ease and flexibility of this feature’s configuration ensure maximum availability and optimum server performance.

The processes are monitored based on rules defined within a configuration file stored on the server. These rules identify criteria for determining the action to be taken if the conditions are met. (For example, if a process becomes unresponsive, the rule could cause the process to be re-spawned.)

Basic parameters pertaining to the Process Monitor were configured as part of the WEM installation or upgrade process. However, additional parameters are available for fine-tuning its configuration to meet the needs of your network.

Important: When installing redundant servers, any WEM services started by the <ems/home>/server/servstart command and monitored by the Process Monitor are not started automatically. Refer to Appendix A for information on installing redundant servers using Oracle Cluster software.

This chapter provides information and procedures for modifying the Process Monitor’s operation using the configuration file.

This chapter includes the following topics:

•	Process Monitor Configuration File

•	Default Rules

•	Verifying the Process Monitor Status

•	Manually Stopping the Process Monitor

•	Manually Starting the Process Monitor

•	Running the Process Monitor as a Stand-alone Application

Important: Unless otherwise specified, all information in this chapter applies to both Sun Solaris- and Red Hat Enterprise Linux-based WEM systems.

Process Monitor Configuration File

Processes are monitored based on rules defined within a plain-text configuration file called psmon.cfg located in the /users/ems/server/etc directory (by default).

Rules are defined within the file using the following syntax:

<Process process_id>

   processing-directives directive_variables

</Process>

The following table provides the syntax descriptions.

psmon.cfg Rule Syntax Descriptions

Syntax	Description
<Process process_id>	Identifies the process to be monitored. The process can be identified by name (i.e. ssh) or by URL (i.e. /users/ems/server/bin/server). Additionally, a wildcard (*) can be specified for monitoring all the processes.
processing-directives directive_variables	Identifies the criteria that must be met before action is taken as well as the action to be taken. The criteria are referred to as “Directives”, while the actions are referred to as “Process Scope Directives”. It is important to note that processing directives can be specified within a specific rule or outside of all rules. When outside, the directive is applied globally to all rules. The psmon.cfg Processing Directives table provides information on the supported processing directives.
</Process>	Indicates the end of a process rule.

‌

psmon.cfg Processing Directives

Directive	Description
Facility	Identifies the syslog facility used for logging. The following facilities are supported: • LOG_KERN • LOG_USER • LOG_MAIL • LOG_DAEMON • LOG_AUTH • LOG_SYSLOG • LOG_LPR • LOG_NEW • SLOG_UUCP • LOG_CRON • LOG_LOCAL0 • LOG_LOCAL1 • LOG_LOCAL2 • LOG_LOCAL3 • LOG_LOCAL4 • LOG_LOCAL5 • LOG_LOCAL6 • LOG_LOCAL7 Default: LOG_DAEMON
LogLevel	Identifies the log level priority used to mark notifications to syslog. The following levels are supported: • LOG_EMERG • LOG_ALERT • LOG_CRIT • LOG_ERR • LOG_WARNING • LOG_NOTICE • LOG_INFO • LOG_DEBUG This directive may also be used in a Process Scope Directive which has precedence over a global declaration. Default: LOG_NOTICE
KillLogLevel (previously KillPIDLogLevel)	Identical to the LogLevel directive, however it only applies to process kill actions. This directive has precedence over the LogLevel Directive and may also be used in a Process Scope Directive which has precedence over a global declaration.
SpawnLogLevel	Identical to the LogLevel directive, however it only applies to process spawn actions. This directive has precedence over the LogLevel Directive and may also be used in a Process Scope Directive which has precedence over a global declaration.
AdminEmail	Specifies the e-mail address to which notification e-mails should be sent. This directive corresponds to the To Email-ID parameter configured during the WEM installation. During the installation, it is stored as a global declaration, however it may also be used in a Process Scope Directive which has precedence over a global declaration. Default: root@localhost
NotifyEmailFrom	Specifies the e-mail address used in the “From” field of sent notification e-mails. Default: <username>@hostname
Frequency	The frequency (measured in seconds) at which the Process Monitor attempts communication with a process. This directive corresponds to the Poll Interval parameter configured during the WEM installation. During the installation, it is stored as a global declaration, however it may also be used in a Process Scope Directive which has precedence over a global declaration. Default: 30 seconds
LastSafePID	Specifies the highest process identification number which the Process Monitor cannot “kill”. When specified, the Process Monitor never attempts to kill a process ID which is numerically less than or equal to the value defined by this directive. This directive is treated as a global directive by default. Default: 800
ProtectSafePIDsQuietly	Enables or disables the suppression of all notifications for preserved process IDs when used in conjunction with the lastsafepid directive. “On” enables this functionality. “Off” disables it. This directive is treated as a global directive by default. Default: Off
SMTPHost	Specifies the IP address or hostname of the Simple Mail Transport Protocol (SMTP) server used for sending e-mail notifications. This directive corresponds to the SMTP Server Name parameter configured during the WEM installation. During the installation, it is stored as a global declaration, however it may also be used in a Process Scope Directive which has precedence over a global declaration. Default: localhost
SMTPTimeout	Specifies the timeout (measured in seconds) used during SMTP connections. Default: 20 seconds
SendmailCmd	Enables the configuration of the sendmail command used for sending e-mail notifications if a failure occurs with the SMTP connection to the host specified by the SMTPHost Directive. Default: /usr/sbin/sendmail -t
Dryrun	Forces this Process Monitor to function as if the --dryrun command line switch had been specified. This can be used for forcing a specific configuration file to only report information but never take any automated action. This directive is treated as a global directive by default. Default: False (disabled)
NotifyDetail	Specifies the verbosity of the notification e-mails. The following levels are supported: • Simple • Verbose • Debug Default: Verbose
PROCESS SCOPE DIRECTIVES
SpawnCmd	Identifies the full command line to be executed in order to re-spawn a dead process.
KillCmd	Identifies the full command line to be executed in order to gracefully shutdown or kill a rogue process. If the command returns a boolean true exit status then, it is assumed that the command failed to execute successfully. If no KillCmd is specified or the command fails, the process is killed by sending a SIGKILL signal with the standard kill() function.
PIDFile	Identifies the full path and filename of a file created by the process containing the identification number of its main parent process.
NUMRETRY	The number of times the Process Monitor attempts to communicate with an un-responsive process before taking action. If the process has not responded to the final attempt within the configured timeout interval, the system considers it unreachable and takes action. This directive corresponds to the Number of Retries parameter configured during the WEM installation. During the installation, it is stored as a Process Scope Directive for each rule defined for WEM process. Default: 10
TMINTVAL	The amount of time (measured in seconds) the system should wait prior to re-attempting to communicate with an un-responsive process. Once the time interval has been reached, the system re-attempts communication for the configured number of retries. This directive corresponds to the Timeout Interval parameter configured during the WEM installation. During the installation, it is stored as a Process Scope Directive for each rule defined for WEM process. Default: 330
TTL	Specifies a maximum time-to-live (in seconds) for a process. The process is killed once it has been running longer than this value, and its process identification number is removed from the defined pidfile.
PctCpu	Specifies the maximum allowable percentage of CPU time a process may use. The process is killed once its CPU usage exceeds this threshold and its process identification number is removed from the defined pidfile.
PctMem	Specifies the maximum allowable percentage of total system memory a process may use. The process will be killed once its memory usage exceeds this threshold and its process identification number is removed from the defined pidfile.
Instances	Specifies the maximum number of instances of a process that are allowed to run simultaneously. The process will be killed once its memory usage exceeds this threshold and its process identification number is removed from the defined pidfile.
NoEmailOnKill	Enables or disables the suppression of e-mail notifications for killed processes. Default: False (disabled)
NoEmailOnSpawn:	Enables or disables the suppression of e-mail notifications for spawned processes. Default: False (disabled)
NoEmail	Enables or disables the suppression of all e-mail notifications. Default: False (disabled)
NeverKillPID	Specifies a list or process identification numbers (separated by spaces) that are never to be killed. Default: 1
NeverKillProcessName	Specifies a list or process names (separated by spaces) that are never to be killed. Default: kswapd kupdated mdrecoveryd

‌

Default Rules

During installation, WEM provided the user with the opportunity to automatically define rules for the following process monitors:

•	EMS Server: Enabled by default

•	Bulkstat Server: Disabled by default

•	Bulkstat Parser: Enabled by default

•	Script Server: Disabled by default

•	Northbound (NB) Server: Disabled by default

•	Notification Service: Disabled by default

Important: Two additional WEM processes are pre-configured to be monitored by the Process Monitor: the Postgres database process and the Apache Webserver process. Configurables for these two processes appear in the psmon.cfg file, but they cannot be altered during the WEM installation process.

The following table identifies the default rules configured for each of the above processes.

Default Rules for WEM Process Monitors Processes

Process	Default Rule
EMS Server	<Process /<ems_dir>/server/bin/server> spawncmd (cd /<ems_dir>/server; /<ems_dir>/server/bin/server) pidfile /<ems_dir>/server/server.pid numretry 10 tmintval 330 </Process>
Bulkstat Server	<Process /<ems_dir>/server/bin/bulkstatserver> spawncmd (cd /<ems_dir>/server; /<ems_dir>/server/bin/bulkstatserver) pidfile /<ems_dir>/server/bsserver.pid numretry 10 tmintval 330 </Process>
Bulkstat Parser	<Process /<ems_dir>/server/bin/bulkstatparser> spawncmd (cd /<ems_dir>/server; /<ems_dir>/server/bin/bulkstatparser) pidfile /<ems_dir>/server/bulkstatparser.pid numretry 10 tmintval 330 </Process>
Script Server	<Process /<ems_dir>/server/bin/scriptsrv> spawncmd (cd /<ems_dir>/server; /<ems_dir>/server/bin/scriptsrv) pidfile /<ems_dir>/server/script.pid numretry 10 tmintval 330 </Process>
Northbound Server	<Process /<ems_dir>/server/bin/nbserver> �spawncmd (cd /<ems_dir>/server; /<ems_dir>/server/bin/nbserver) pidfile /users/ems/server/nbserver.pid numretry 10 tmintval 330 </Process>
Notify Service	<Process /<ems_dir>/server/bin/Notify_Service> spawncmd(cd /<ems_dir>/server; /<ems_dir>/server/bin/nbSrvr pidfile /<ems_dir>/server/notify_service.pid numretry 10 tmintval 330 </Process>
Postgres Database	<Process /<ems_dir>/postgresx.x.x/bin/postmaster -i> spawncmd /<ems_dir>/server/scripts/postgresctl start pidfile /<ems_dir>/postgresx.x.x/data/postmaster.pid numretry 10 tmintval 330 </Process>
Apache Webserver	<Process /<ems_dir>/apache/bin/httpd -f /<ems_dir>/apache/conf/httpd.conf> spawncmd /<ems_dir>/apache/bin/apachectl start pidfile /<ems_dir>/apache/logs/httpd.pid numretry 10 tmintval 330 </Process>

‌

Verifying the Process Monitor Status

The status of the Process Monitor can be checked at any time by executing either of the following commands:

ps -ef | grep psmon

or

./serv monitor

The first command indicates whether or not an active psmon process is running. The second command performs one of the following:

•	If the Process Monitor is stopped, executing this command is equivalent to executing the ./serv monitor start command (refer to the Manually Starting the Process Monitor section in this chapter).

•	If the Process Monitor is currently running, a message is displayed indicating that it is running. A prompt is also provided that allows you to restart the process. To restart the process, enter yes.

Manually Stopping the Process Monitor

Upon installation of the WEM, the Process Monitor is started automatically. This section provides instructions for manually disabling it. This can be useful if changes are made to the configuration file.

Follow the instructions below to manually stop the Process Monitor.

Step 1

Login as the root user.

Step 2

Go to the directory in which the WEM Server application file is located. By default, this is the /<ems_dir>/server directory. Enter the following command:

cd /<ems_dir>/server

Step 3

Stop the Process Monitor by entering the following command:

./serv monitor stop

Step 4

Verify that the Process Monitor has stopped by executing the following command:

ps -ef | grep psmon

If the Process Monitor was successfully stopped, this command finds no active process and returns no result.

Manually Starting the Process Monitor

Upon installation of the WEM, the Process Monitor is started automatically. However, if the process was stopped, it can be started using the information and instructions in this section.

Step 1

Login as the root user.

Step 2

Go to the directory in which the WEM Server application file is located. By default, this is the /<ems_dir>/server directory. Enter the following command:

cd /<ems_dir>/server

Step 3

Start the Process Monitor by entering the following command:

./serv monitor start

Once the Process Monitor is started, a status message is displayed. The process identification number assigned to the psmon process, and the directory in which the created log file is located is also displayed.

Running the Process Monitor as a Stand-alone Application

As mentioned previously, the Process Monitor can be run as a background deamon (the default operation when enabled during the installation of the WEM) or a stand-alone application.

This section provides information and instructions for running the Process Monitor as a stand-alone application from the command line interface.

To run the Process Monitor from the command line, use the following instructions:

Step 1

Login as the root user.

Step 2

Go to the directory in which the Process Monitor application file is located. By default, this is the /<ems_dir>/server directory. Enter the following command:

cd /<ems_dir>/server

Step 3

Launch the Process Monitor application by entering the following command:

./psmon [--conf=filename] [--daemon] [--cron] [--user=user] [--adminemail=emailaddress] [--dryrun] [--help] [--version]

Keyword/Variable	Description
--conf=filename	Specify an alternate configuration file name (other than the psmon.cfg file).
--daemon	Start as background deamon.
--cron	Disables already running errors when trying to launch (i.e. with the --daemon option).
--user=user	Specifies that only processes running under the specified username should be scanned.
--adminemail=emailaddress	Specifies the e-mail address to send notifications to.
--dryrun	Provides notifications but does not kill or spawn new processes.
--help	Displays the supported keywords.
--version	Displays the version information.

‌

Depending on the command used, the rules dictated by the configuration file are executed.

 

 

Cisco Systems Inc.

Tel: 408-526-4000

Fax: 408-527-0883